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Abstract 

Recently, many results on the computational complexity of sorting algorithms were ob- 
tained using Kolmogorov complexity (the incompressibility method). Especially, the usually 
hard average-case analysis is ammenable to this method. Here we survey such results about Bub- 
blesort, Heapsort, Shellsort, Dobosiewicz-sort, Shakersort, and sorting with stacks and queues in 
sequential or parallel mode. Especially in the case of Shellsort the uses of Kolmogorov complex- 
ity surprisingly easily resolved problems that had stayed open for a long time despite strenuous 
attacks. 

1 Introduction 

We survey recent results in the analysis of sorting algorithms using a new technical tool: the 
incompressibility method based on Kolmogorov complexity. Complementing approaches such as the 
counting method and the probabilistic method, the new method is especially suited for the average- 
case analysis of algorithms and machine models, whereas average-case analysis is usually more 
difficult than worst-case analysis using the traditional methods. Obviously, the results described 
can be obtained using other proof methods — all true provable statements must be provable from 
the axioms of mathematics by the inference methods of mathematics. The question is whether a 
particular proof method facilitates and guides the proving effort. The following examples make 
clear that thinking in terms of coding and the incompressibility method suggests simple proofs 
that resolve long-standing open problems. A survey of the use of the incompressibility method 
in combinatorics, computational complexity, and the analysis of algorithms is [16] Chapter 6, and 
other recent work is [2, 15]. 

We give some definitions to establish notation. For introduction, details, and proofs, see [16]. 
We write string to mean a finite binary string. Other finite objects can be encoded into strings in 
natural ways. The set of strings is denoted by {0, 1}*. Let x,y, z G M, where M denotes the set of 
natural numbers. Identify J\f and {0, 1}* according to the correspondence 

(0,e), (1,0), (2,1), (3, 00), (4, 01),.... 

Here e denotes the empty word with no letters. The length of x is the number of bits in the binary 
string X and is denoted by l{x). For example, Z(OIO) = 3 and /(e) = 0. The emphasis is on binary 
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sequences only for convenience; observations in every alphabet can be so encoded in a way that is 
'theory neutral'. 

Self-delimiting Codes: A binary string y is a proper prefix of a binary string x if we can 

write X = yz for z ^ e. A set {x, y, ■ ■ ■} ^ {0, 1}* is prefix-free if for every pair of distinct elements 
in the set neither is a proper prefix of the other. A prefix-free set is also called a prefix code. Each 
binary string x = x\X2 ■ ■ - Xn has a special type of prefix code, called a self-delimiting code, 

X = l"0a:;ia;2 . . . x„. 

This code is self-delimiting because we can effectively determine where the code word x ends 
by reading it from left to right without backing up. Using this code we define the standard self- 
delimiting code for x to be x' = l{x)x. It is easy to check that l{x) = 2n-\-l and l{x') = n-\-2 log n-\-l. 

Let (•, •) be a standard one-one mapping from Af x J\f to J\f, for technical reasons chosen such 
that l{{x,y)) = l{y) + l{x) + 2l{l{x)) + 1, for example (x,y) = x'y = l'('(^))OZ(x)xy. 

Kolmogorov Complexity: Informally, the Kolmogorov complexity, or algorithmic entropy, 
C (x) of a string x is the length (number of bits) of a shortest binary program (string) to compute x 
on a fixed reference universal computer (such as a particular universal Turing machine) . Intuitively, 
C{x) represents the minimal amount of information required to generate x by any effective process, 
[10]. The conditional Kolmogorov complexity C(a: | y) of a; relative to y is defined similarly as the 
length of a shortest program to compute x, if y is furnished as an auxiliary input to the compu- 
tation. The functions C(-) and C(- | •), though defined in terms of a particular machine model, 
are machine-independent up to an additive constant (depending on the particular enumeration of 
Turing machines and the particular reference universal Turing machine selected). They acquire an 
asymptotically universal and absolute character through Church's thesis, and from the ability of 
universal machines to simulate one another and execute any effective process, see for example [16]. 
Formally: 

Definition 1 Let Tq, Ti, . . . be a standard enumeration of all Turing machines. Choose a universal 
Turing machine U that expresses its universality in the following manner: 

U{{{i,p),y))=Ti{{p,y)) 

for all i and {p,y), where p denotes a Turing program for Tj and y an input. We fix U as our 
reference universal computer and define the conditional Kolmogorov complexity of x given y by 

C{x I y) = min {l{q) : U{{q,y)) = x}, 
qe{o,i}* 

for every q (for example q = {i,p) above) and auxiliary input y. The unconditional Kolmogorov 
complexity of x is defined by C{x) = C{x \ e). For convenience we write C{x,y) for C{{x,y)), and 
C{x I y,z) for C(x | {y,z)). 

Incompressibility: First we show that the Kolmogorov complexity of a string cannot be signif- 
icantly more than its length. Since there is a Turing machine, say Tj, that computes the identity 
function Ti(x) = x, and by definition of universality of U we have U{{i,p)) = Ti{p). Hence, 
C{x) < l{x) -\- c for fixed c < 2 log z -|- 1 and all x. ^ 

^ "2 log i" and not "log i" since we need to encode i in such a way that U can determine the end of the encoding. 
One way to do that is to use the code {l{i))Ol{i)i which has length 2l{l(i)) -\- -|- 1 < 21ogi bits. 
^In what follows, "log" denotes the binary logarithm. "|^rj" is the greatest integer q such that q < r. 
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It is easy to see that there are also strings that can be described by programs much shorter than 
themselves. For instance, the function defined by /(I) = 2 and f{i) = 2f^'^~^^ for z > 1 grows very 
fast, f{k) is a "stack" of k twos. It is clear that for every k it is the case that f{k) has complexity 
at most C{k) + 0(1). What about incompressibility? For every n there are 2"^ binary strings of 
length sn, but only J22=Q 2* = 2" — 1 descriptions in binary string format of lengths less than n. 
Therefore, there is at least one binary string x of length n such that C{x) > n. We call such strings 
incompressible. The same argument holds for conditional complexity: since for every length n 
there arc at most 2" — 1 binary programs of lengths < n, for every binary string y there is a binary 
string X of length n such that C{x \ y) > n. Strings that are incompressible are patternless, since 
a pattern could be used to reduce the description length. Intuitively, we think of such patternless 
sequences as being random, and we use "random sequence" synonymously with "incompressible 
sequence." There is also a formal justification for this equivalence, which does not need to concern 
us here. Since there are few short programs, there can be only few objects of low complexity: the 
number of strings of length n that are compressible by at most 8 bits is at least 2" — 2""'^ + 1. 

Lemma 1 Let 6 be a positive integer. For every fixed y, every set S of cardinality m has at least 
m{l — 2^^) + 1 elements x with C{x\y)> [logmj — 6. 

Proof. There are N = YI^Zq 2* = 2" — 1 binary strings of length less than n. A fortiori there 
are at most N elements of S that can be computed by binary programs of length less than n, given 
y. This implies that at least m — N elements of S cannot be computed by binary programs of 
length less than n, given y. Substituting n by [log m\ — 5 together with Definition 1 yields the 
lemma. □ 



Lemma 2 If A is a set, then for every y every element x & A has complexity C(x\A, y) < log |^| + 
0(1). 

Proof. A string x £ A can be described by first describing A in 0(1) bits and then giving the 
index of x in the enumeration order of ^. □ 

As an example, set S = {x : l{x) = n}. Then is |5| =2". Since C{x) < n + c for some fixed c 
and all x in S, Lemma 1 demonstrates that this trivial estimate is quite sharp. If we arc given S as 
an explicit tabic then we can simply enumerate its elements (in, say, lexicographical order) using 
a fixed program not depending on S or y. Such a fixed program can be given in 0(1) bits. Hence 
the complexity satisfies C{x \ S,y) < loglS*] + 0(1). 

Incompressibility Method: In a typical proof using the incompressibility method, one first 
chooses an incompressible object from the class under discussion. The argument invariably says 
that if a desired property does not hold, then in contrast with the assumption, the object can 
be compressed significantly. This yields the required contradiction. Since most objects are almost 
incompressible, the desired property usually also holds for almost all objects, and hence on average. 
Below, we demonstrate the utility of the incompressibility method to obtain simple and elegant 
proofs. 

Average-case Complexity: For many algorithms, it is difficult to analyze the average-case 
complexity. Generally speaking, the difficulty comes from the fact that one has to analyze the 
time complexity for all inputs of a given length and then compute the average. This is a difficult 
task. Using the incompressibility method, we choose just one input — a representative input. 
Via Kolmogorov complexity, we can show that the time complexity of this input is in fact the 
average-case complexity of all inputs of this length. Constructing such a "representative input" is 
impossible, but we know it exists and this is sufficient. 
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In average-case analysis, the incompressibility method has an advantage over a probabihstic 
approach. In the latter approach, one deals with expectations or variances over some ensemble of 
objects. Using Kolmogorov complexity, we can reason about an incompressible individual object. 
Because it is incompressible it has all simple statistical properties with certainty, rather than having 
them hold with some (high) probability as in a probabilistic analysis. This fact greatly simplifies 
the resulting analysis. 



A simple introductory example of the application of the incompressibility method is the average-case 
analysis of Bubblesort. The classical approach can be found in [11]. It is well-known that Bubblesort 
uses 0(n^) comparisons/exchanges on the average. We present a very simple proof of this fact. 
The proof is based on the following intuitive idea: There arc n! diff'crcnt permutations. Given the 
sorting process (the insertion paths in the right order) one can recover the correct permutation 
from the sorted list. Hence one requires n\ pairwise different sorting processes. This gives a lower 
bound on the minimum of the maximal length of a process. We formulate the proof in the crisp 
format of incompressibility. In Bubblesort we make passes from left to right over the permutation 
to be sorted and always move the currently largest element right by exchanges between it and the 
right-adjacent element — if that one is smaller. We make at most n — 1 passes, since after moving all 
but one element in the correct place the single remaining element must be also in its correct place (it 
takes two elements to be wrongly placed). The total number of exchanges is obviously at most ra^, 
so we only need to consider the lower bound. Let B be a Bubblesort algorithm. For a permutation 
TT of the elements 1, . . . , n, we can describe the total number of exchanges by M := J27=i where 
rui is the initial distance of element n — i to its final position. Note that in every pass more than one 
element may "bubble" right but that means simply that in the future passes of the sorting process 
an equal number of exchanges will be saved for the element to reach its final position. That is, 
every element executes a number of exchanges going right that equals precisely the initial distance 
between its start position to its final position. It is clear that M <n^ for all permutations. Given 
mi, . . . , rrin-i, in that order, we can reconstruct the original permutation from the final sorted list. 
Since choosing a elements from a list of 6 -|- a elements divides the remainder in a sequence of a -|- 1 
possibly empty sublists, there are 



possibilities to partition M into n — 1 ordered non-negative summands. Therefore, we can describe 
TT by M, n, an index of log B[M) bits to describe mi, . . . , m„_i among all partitions of M, and an 
program P that reconstructs vr from these parameters and the final sorted list 1, . . . ,n. Consider 
permutations tt satisfying C(7r | n,B{M),P) > logn! — logn. Then by Lemma 2 at least a 
(1 — l/ri)th fraction of all permutations of n elements have that high complexity. Under this 
complexity condition on tt, we also have M > n. (If M < n then C{tt \ n, B{M), P) = 0{n).) Since 
the description of tt we have constructed is effective, its length must be at least C(7r | n,B,P). 
Encoding M self-delimiting, in order to be able to separate M from B{M) in a concatenation of the 
binary descriptions, we therefore find log M+2 log log M-|- log S(M) > nlogn— O(logn). Substitute 
a good estimate for log i?(M) (the formula used later in the Shellsort example, Section 4) divide by 
n, and discard the terms that vanish with n, assuming 2 < n < M < ra^, yields log(H-M/ (ra — 2)) > 
logn-|- 0(1). By the above, this holds for at least an (1 — l/n)th fraction of all permutations, and 
hence gives us an ^{p?) lower bound on the expected number of comparisons/exchanges. 



2 Bubblesort 
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3 Heapsort 



Heapsort is a widely used sorting algorithm. One reason for its prominence is that its running time 
is guaranteed to be of order nlogn, and it does not require extra memory space. The method was 
first discovered by J.W.J. Williams, [29], and subsequently improved by R.W. Floyd [4] (see [11]). 
Only recently has one succeeded in giving a precise analysis of its average-case performance [23]. 
I. Munro has suggested a remarkably simple solution using incompressibility [18] initially reported 
in [16]. 

A "heap" can be visualized as a complete directed binary tree with possibly some rightmost 
nodes being removed from the deepest level. The tree has n nodes, each of which is labeled with 
a different key, taken from a linearly ordered domain. The largest key ki is at the root (on top of 
the heap), and each other node is labeled with a key that is less than the key of its father. 

Definition 2 Let keys be elements of J\f. An array of keys fci, . . . , A;„ is a heap if they are partially 
ordered such that 

klj/2\ > kj for 1 < [j/2\ <j<n. 

Thus, ki > k2,ki > k^,k2 > k^, and so on. We consider "in place" sorting of n keys in an array 
yl[l..n] without use of additional memory. 

Heapsort {Initially, 74[l..n] contains n keys. After sorting is completed, the keys in A will be 
ordered as ^[1] < A[2] < ■ ■ ■ < A[n].} 

Heapify: {Regard A as a tree: the root is in A[l]; the two sons of A[i] arc at A[2i] and A[2i + 1], 
when 2i,2i + 1 < n. We convert the tree in ^4 to a heap.} Repeat for i = [n/2j, [n/2\ — 
1, . . . , 1: {the subtree rooted at A[i] is now almost a heap except for A[i]} push the key, say 
k, at A[i] down the tree (determine which of the two sons of A[i] possesses the greatest key, 
say k' in son A[2i + j] with j equals or 1); if k' > k then put k in A[2i + j] and repeat 
this process pushing k' at A[2i + j] down the tree until the process reaches a node that does 
not have a son whose key is greater than the key now at the father node. 

Sort: Repeat for i = n,n — l, . . . ,2: {A[l..i] contains the remaining heap and A[i + l..n] contains 
the already sorted list ki+i, . . . ,kn of largest elements. By definition, the element on top of 
the heap in ^[1] must be fcj.} switch the key ki in A[l] with the key k in A[i], extending the 
sorted list to A[i..n]. Rearrange A[l..i — 1] to a heap with the largest element at ^[1]. 

It is well known that the Heapify step can be performed in 0{n) time. It is also known that the 
Sort step takes no more than 0(n log n) time. Wc analyze the precise average-case complexity of 
the Sort step. There are two ways of rearranging the heap: Williams's method and Floyd's method. 

Williams's Method: {Initially, A[l] = k.} 

Repeat compare the keys of fc's two direct descendants; if m is the larger of the two then compare 
k and m; if k < m then switch k and m in A[l..i — 1] until k > m. 

Floyd's Method: {Initially, A[l] is empty.} Set j := 1; 

while A[j] is not a leaf do: 

if A[2j] > A[2j + 1] then j := 2j 
else j := 2j + 1; 
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while k > A[j] do: 

{back up the tree until the correct position for k} j := [j/2j; 

move keys of A[j] and each of its ancestors one node upwards; 
Set A\j] := k. 

The difference between the two methods is as follows. Williams's method goes from the root at the 

top down the heap. It makes two comparisons with the son nodes and one data movement at each 
step until the key k reaches its final position. Floyd's method first goes from the root at the top 
down the heap to a leaf, making only one comparison each step. Subsequently, it goes from the 
bottom of the heap up the tree, making one comparison each step, until it finds the final position 
for key k. Then it moves the keys, shifting every ancestor of k one step up the tree. The final 
positions in the two methods are the same; therefore both algorithms make the same number of key 
movements. Note that in the last step of Floyd's algorithm, one needs to move the keys carefully 
upward the tree, avoiding swaps that would double the number of moves. 

The heap is of height logn. If Williams's method uses 2d comparisons, then Floyd's method uses 
d + 25 comparisons, where 5 = log n — d. Intuitively, 5 is generally very small, since most elements 
tend to be near the bottom of the heap. This makes it likely that Floyd's method performs better 
than Williams's method. We analyze whether this is the case. Assume a uniform probability 
distribution over the lists of n keys, so that all input lists are equally likely. 

Average-case analysis in the traditional manner suffers from the problem that, starting from a 
uniform distribution on the lists, it is difficult to compute the distribution on the resulting initial 
heaps, and increasingly more difficult to compute the distributions on the sequence of decreasing- 
size heaps after subsequent heapsort steps. The sequence of distributions seem somehow realated, 
but this is hard to express and exploit in the traditional approach. In contrast, using Kolmogorov 
complexity we express this similarity without having to be precise about the distributions. 

Theorem 1 On average (uniform distribution), Heapsort makes nlogn + 0{n) data movements. 
Williams's method makes 2n log n — 0(n) comparisons on average. Floyd's method makes nlogn + 
0(n) comparisons on average. 

Proof. Given n keys, there are nl (« n"'e~"'V27m by Stirling's formula) permutations. Hence 
we can choose a permutation p of n keys such that 

C{p\n) > nlogn — 2n, (1) 

justified by Theorem 1, page 3. In fact, most permutations satisfy Equation 1. 

Claim 1 Let h be the heap constructed by the Heapify step with input p that satisfies Equation 1. 
Then 

C{h\n) > nlogn — 6n. (2) 

Proof. Assume the contrary, C{h\n) < nlogn — 6n. Then we show how to describe p, 
using h and n, in fewer than n log n — 2n bits as follows. We will encode the Heapify process 
that constructs h from p. At each loop, when we push k = A[i] down the subtree, we record 
the path that key k traveled: indicates a left branch, 1 means a right branch, 2 means halt. 
In total, this requires (n log 3) j/2-'^^ < 2nlog3 bits. Given the final heap h and the above 
description of updating paths, we can reverse the procedure of Heapify and reconstruct p. Hence, 
C{p\n) < C{h\n) + 2nlog3 + 0(1) < nlogn — 2n, which is a contradiction. (The term 6n above 
can be reduced by a more careful encoding and calculation.) □ 
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We give a description of h using the history of the n — 1 heap rearrangements during the Sort 
step. We only need to record, for z := n — 1, . . . , 2, at the {n — i + l)st round of the Sort step, the 
final position where A[i] is inserted into the heap. Both algorithms insert A[i] into the same slot 
using the same number of data moves, but a different number of comparisons. 

We encode such a final position by describing the path from the root to the position. A path can 
be represented by a sequence s of O's and I's, with indicating a left branch and 1 indicating a right 
branch. Each path i is encoded in self-delimiting form by giving the value 6i = log n — l{si) encoded 
in self-delimiting binary form, followed by the literal binary sequence Sj encoding the actual path. 
This description requires at most 

l{si) + 2logSi (3) 
bits. Concatenate the descriptions of all these paths into sequence H. 

Claim 2 We can effectively reconstruct heap h from H and n. 

Proof. Assume H is known and the fact that /i is a heap on n different keys. Wc simulate the 
Sort step in reverse. Initially, ^[l..n] contains a sorted list with the least element in ^[1]. 

for i := 2, . . . , n — 1 do: {now — 1] contains the partially constructed heap and ^[i..n] con- 

tains the remaining sorted list with the least element in A[i]} Put the key of A[i] into A[l], 
while shifting every key on the (n — i)th path in H one position down starting from the root 
at ^[1]. The last key on this path has nowhere to go and is put in the empty slot in A[i]. 

termination {Array ^[l..n] contains heap h.} 

□ 

It follows from Claim 2 that C{h\n) < 1{H) + 0(1). Therefore, by Equation 2, we have 1{H) > 
nlogn — 6n. By the description in Equation 3, we have 

n n 

^{l{si) + 2\og5i) = ^((logn) - 5i -I- 2 log (5,) > nlogn - 6n. 

i=l i=l 

It follows that Er=i('^j - 21og5i) < 6n. This is only possible if ELi = 0{n). Therefore, the 

average path length is at least logn — c, for some fixed constant c. In each round of the Sort 
step the path length equals the number of data moves. The combined total path length is at least 
n log n — nc. 

It follows that starting with heap h, Heapsort performs at least nlogn — 0(n) data moves. 
Trivially, the number of data moves is at most n log n. Together this shows that Williams's method 
makes 2nlogn — 0(n) key comparisons, and Floyd's method makes nlogn-l-O(n) key comparisons. 

Since most permutations are Kolmogorov random, these bounds for one random permutation p 
also hold for all permutations on average. But we can make a stronger statement. We have taken 
C{p\n) at least en below the possible maximum, for some constant e > 0. Hence, a fraction of at 
least 1 — 2""^" of all permutations on n keys will satisfy the above bounds. □ 

4 Shellsort 

The question of a nontrivial general lower bound (or upper bound) on the average complexity of 
Shellsort (due to D.L. Shell [26]) has been open for about four decades [11, 25], and only recently 
such a general lower bound was obtained. The original proof using Kolmogorov complexity [12] is 



7 



presented here. Later, it turned out that the argument can be translated to a counting argument 
[13]. It is instructive that thinking in terms of code length and Kolmogorov complexity enabled 
advances in this problem. 

Shcllsort sorts a list of n elements in p passes using a sequence of increments hi, . . . ,hp. In 
the fcth pass the main list is divided in separate sublists of length \n/hk\, where the ith sublist 
consists of the elements at positions j, where j mod hk = i— 1, of the main list {i = 1,. . . , hk)- Every 
sublist is sorted using a straightforward insertion sort. The efficiency of the method is governed 
by the number of passes p and the selected increment sequence hi, . . . ,hp with hp = 1 to ensure 
sortedness of the final list. The original logn-pass ^ increment sequence [n/2j , [n/4j , . . . , 1 of Shell 
[26] uses worst case 0(n^) time, but Papernov and Stasevitch [19] showed that another related 
sequence uses 0(n^/^) and Pratt [22] extended this to a class of all nearly geometric increment 
sequences and proved this bound was tight. The currently best asymptotic method was found by 
Pratt [22]. It uses all log^ n increments of the form 2*3-' < [n/2j to obtain time 0(n log^ n) in the 
worst case. Moreover, since every pass takes at least n steps, the average complexity using Pratt's 
increment sequence is B(nlog^n). Incerpi and Sedgewick [5] constructed a family of increment 
sequences for which Shcllsort runs in (9(n^+'"/ V^°S"^ time using (8/e^) logn passes, for every e > 0. 
B. ChazcUc (attribution in [24]) obtained the same result by generalizing Pratt's method: instead of 
using 2 and 3 to construct the increment sequence use a and (a + 1) to obtain a worst-case running 

time of n log^ n(a^/ In^ a) which is 0(n^^''/'^^°^'*) for In^ a = O(logn). Poonen [20], and Plaxton, 
Poonen and Suel [21], demonstrated an Q(n}'^^/^) lower bound for p passes of Shellsort using 
any increment sequence, for some e > 0; taking p = O(logn) shows that the Incerpi-Sedgewick / 
Chazelle bounds are optimal for small p and taking p slightly larger shows a 0(n log^ n/(log log n)^) 
lower bound on the worst-case complexity of Shellsort. For the average-case running time Knuth 
[11] showed 9(n^/^) for the best choice of increments in p = 2 passes; Yao [30] analyzed the average- 
case for p = 3 but did not obtain a simple analytic form; Yao's analysis was improved by Janson and 
Knuth [7] who showed 0(n^'^/^^) average-case running time for a particular choice of increments in 
p = 3 passes. Apart from this no nontrivial results are known for the average-case; see [11, 24, 25]. 
In [12, 13] a general D,{pn^'^^/P) lower bound was obtained on the average-case running time of 
p-pass Shellsort under uniform distribution of input permutations, for every l<p< n/2. ^ This 
is the first advance on the problem of determining general nontrivial bounds on the average-case 
running time of Shellsort [22, 11, 30, 5, 21, 24, 25]. 

A Shellsort computation consists of a sequence of comparison and inversion (swapping) oper- 
ations. In this analysis of the average-case lower bound we count just the total number of data 
movements (here inversions) executed. The same bound holds a fortiori for the number of compar- 
isons. 

Theorem 2 The average number of comparisons (and also inversions for p = o{\ogn)) in p-pass 
Shellsort on lists of n keys is at least Q, (^pn^^^^^^ for every increment sequence. The average is 
taken with all lists of n items equally likely (uniform distribution). 

Proof. Let the list to be sorted consist of a permutation vr of the elements 1, . . . , n. Consider 
a [hi, . . . ,hp) Shellsort algorithm A where h^ is the increment in the /cth pass and hp = 1. For 
every 1 < i < n and 1 < < p, let rui^k be the number of elements in the /i^-increment sublist, 
containing element i, that are to the left of i at the beginning of pass k and are larger than i. 

^ "log" denotes the binary logarithm and "In" denotes the natural logarithm. 

*The trivial lower bound is f2(pn) comparisons since every element needs to be compared at least once in every 
pass. 
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Observe that J2^=i f^i,k is the number of inversions in the initial permutation of pass fc, and that 
the insertion sort in pass k requires precisely Z]r=i("^i,fc + 1) comparisons. Let M denote the total 
number of inversions: 

P n 

M ■.= Y,Y.^i,k- (4) 

fe=l i=l 

Claim 3 Given all the rrii^k 's in an appropriate fixed order, we can reconstruct the original permu- 
tation TT. 

Proof. In general, given the mj^^'s and the final permutation of pass k, we can reconstruct 
the initial permutation of pass k. □ 

Let M as in (4) be a fixed number. There are n! permutations of n elements. Let permutation 
TT be an incompressible permutation having Kolmogorov complexity 

C{Tr\n,A,P)>\ogn\-\ogn, (5) 

where P is the decoding program in the following discussion. There exist many such permutations 
by lemma 1. Clearly, there is a fixed program that on input A,P,n reconstructs tt from the 
description of the nii^kS as in Claim 3. Therefore, the minimum length of the latter description, 
including a fixed program in 0(1) bits, must exceed the complexity of tt: 

C(mi,i, . . . , mn,p\n, A, P) + 0(1) > C(7r|n, A, P). (6) 

An M as defined by (4) such that every division of M in mj^fe's contradicts (6) would be a lower 
bound on the number of inversions performed. Similar to the reasoning Bubblesort example. Sec- 
tion 2, there are 

D(M).= {"^-'>-'] (7) 

\ np — 1 I 

distinct divisions of M into np ordered nonnegative integral summands rrii^kS. Every division can 
be indicated by its index j in an enumeration of these divisions. This is both obvious and an 
application of lemma 2. Therefore, a description of M followed by a description of j effectively 
describes the rrii^kS. Fix P as the program for the reference universal machine that reconstructs 
the ordered list of irii^kS from this description. The binary length of this two-part description must 
by definition exceed the Kolmogorov complexity of the described object. 

A minor complication is that we cannot simply concatenate two binary description parts: the 
result is a binary string without delimiter to indicate where one substring ends and the other one 
begins. Encoding the M part of the description self-delimitingly we obtain: 

log D{M) + log M + 2 log log M + 1 > C(mi,i, . . . , mn,p\n, A, P). 

We know that M < pn^ since every rui^k < n. We can assume^ p < n. Together with (5) and (6), 
we have 

logD(M) > logn! - 41ogn- 21oglogn- 0(1). (8) 



^Otherwise we require at least comparisons. 
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Estimate logL»(M) by ^ 
, fM + np-l\ , M + np-1 M + - 1 1, M + np - 1 

The second term in the right-hand side equals^ 

log (l + ^)'''<loge"^-i 

for all positive M and — 1 > 0. Since < p < n and n < M < pn^, 

1 , M + nn - 1 

log > 

2{np - 1) (np - 1)M 

for n — GO. Therefore, log£)(M) is majorized asymptotically by 

{np - 1) (log (^^^^ + l) + log 

for n — GO. Since the righthand-side of (8) is asymptotic to nlogn for n — oo, this yields 

M = ^{pr?+^'P), 

for p = o(logn). (More precisely, M = Q.ijm}'^^^^'^^/^) for p < (e/loge) logn (0 < e < 1), see [13].) 
That is, the running time of the algorithm is as stated in the theorem for every permutation vr 
satisfying (5). By lemma 1 at least a (1 — l/n)-fraction of all permutations vr require that high 
complexity. Then the following is a lower bound on the expected number of inversions of the sorting 
procedure: 

(1 - -)n(pn^+^^p) + -n(o) = n(pn^+^/p), 

n n 

for p = o(logn). For p = ^(logn), the lower bound on the number of comparisons is trivially 
pn = n{pn^+yp). This gives us the theorem. □ 

Our lower bound on the average-case can be compared with the Plaxton-Poonen-Suel Q{n^~^''^ ^) 
worst case lower bound [21]. Some special cases of the lower bound on the average-case complexity 
are: 

1. For p = 1 our lower bound is asymptotically tight: it is the average number of inversions for 
Insertion Sort. 

2. For p = 2, Shellsort requires ft{n^^^) inversions (the tight bound is known to be Q{n^^^) [11]); 

3. For p = 3, Shellsort requires ^(n^/^) inversions (the best known upper bound is 0(n^^/^^) in 

[7]); 

®Use the following formula ([16], p. 10), 

log y = 6 log - + (a - 6) log ^ + - log + O ( 1 ) . 

^Use > (1 + f for all a > and positive integer b. 



10 



4. For p = log n/ log log n, Shellsort requires Q,{n log^ n/ log log n) inversions; 

5. For p = logn, Shellsort requires Vl{n\ogn) comparisons on average. This is of course the 
lower bound of average number of comparisons for every sorting algorithm. 

6. In general, for n/2 > p = p{n) > logn, Shellsort requires ^{n-p{n)) comparisons (since every 
pass trivially makes n comparisons). 

In [25] it is mentioned that the existence of an increment scqTicncc yielding an average O(nlogn) 
Shellsort has been open for 30 years. The above lower bound on the average shows that the 
number p of passes of such an increment sequence (if it exists) is precisely p = O(logn); all the 
other possibilities are ruled out: Is there an increment sequence for log n-pass Shellsort so that it 
runs in average-case 0(nlogn)? Can we tighten the average-case lower bound for Shellsort? The 
above bound is known to be not tight for p = 2 passes. 

5 Dobosiewicz Sort and Shakersort 

We look at some variants of Shellsort. Knuth [11], 1st Edition Exercise 5.2.1.40 on page 105, and 
Dobosiewicz [3] proposed to use only one pass of Bubblesort on each subsequence instead of sorting 
the subsequences at each stage. Incerpi and Sedgewick [6] used two passes of Bubblesort in each 
stage, one going left-to-right and the other going right-to-left. This is called Shakersort since it 
reminds one of shaking a cocktail. In both cases the sequence may stay unsorted, even if the last 
increment is 1. A final phase, a straight insertion sort, is required to guaranty a fully sorted list. 
Until recently, these variants have not been seriously analyzed; in [3, 6, 28, 24] mainly empirical 
evidence is reported, giving evidence of good running times (comparable to Shellsort) on randomly 
generated input key sequences of moderate length. The evidence also suggests that the worst-case 
running time may be quadratic. Again, let n be the number of keys to be sorted and let p be 
the number of passes. The U{n^~^^^^) worst-case lower bound of Poonen [20] holds apart from 
Shellsort also for the variants of it. We also have a worst-case lower bound of f2(n^) on Dobosiewicz 
sort and Shaker sort for the special case of almost geometric sequences of increments. But recently 
Brcjova [1] proved that Shaker sort runs in 0(n^/^ log^ n) worst-case time for a certain sequence of 
increments (the first non-quadratic worst-case upper bound). Using the incompressibility method, 
she also proved lower bounds on the average-case running times. 

Theorem 3 There is an ^}{n'^/4^) lower bound on the average-case running time of Shaker sort, 
and a 0(^^/2^) lower bound on the average-case running time of Dobosiewicz sort. The avereges 
are taken with respect to the uniform distribution. 

Remark 1 These lower bounds (on the average-case) are better than the Poonen [20] lower bounds 
of 0(77.^"''^/ v^) on the worst-case. 

Proof. Consider Dobosiewicz sorting algorithm A (the description of A includes the number 
of passes p and the list of increments hi, ... , hp). Every comparison based sorting algorithm uses 
Q{nlogn) comparisons on average. U p > logn — log logn then the claimed lower bound trivially 
holds. So we can assume that p < logn — log logn. Let tt be the permutation of {0, 1, . . . , n — 1} 
to be sorted, and let vr' be the permutation remaining after all p stages of the Dobsiewicz sort, but 
before the final insertion sort. If X is the number of inversions in tt' then the final insertion sort 
takes fl{X) time. 
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Claim 4 Let tt be a permutation satisfying (5). Then X = Q.{p? /2P). 

Proof. We can reconstruct tt from vr' given p strings of lengths n defined as follows: The jth 
bit of the ith string is "1" if Xj was interchanged with Xj^h^ in the ith phase of the algorithm {hi 
is the ith increment), and "0" otherwise. Given tt' and these strings in appropriate order we can 
simply run the p sorting phases in reverse. 

Furthermore, tt' can be reconstructed from its inversion table oq, a,2, ■ ■ ■ , a„_i, where is the 
number of elements in list vr' left of the ith position that are greater than the element in the ith 
position. Thus, J^i '^i = There are D[X) = {^^^^^) ordered partitions of X into n non-negative 
summands. Hence, tt' can be reconstructed from X and an index of \o%D{X) bits identifying 
the partition in question. Given n, we encode X self-delimiting to obtain a total description of 
log X + 2 log log X + \ogD{X) bits. 

Therefore, with P the reconstruction program, we have shown that 

C(7r I n. A, P) <np + \ogX + 2 log log X + log D{X). 
Estimating asymptotically, similar to the part following (8), 

\ogD{X) < (n - 1) log + l) + 0{n). 

Since vr satisfies (5), we have np + {n — 1) log (^^^^ + 1^ + 0{n) > nlogn — @{n). Hence, X > 

n^/(2^')6(l) = Q,{n'^/2P), where the last equality holds sincep < log ri — log log and hence n^/2P > 
n log n. □ 

By lemma 1 at least a (1 — l/n)-fraction of all permutations vr require that high complexity. 
This shows that the running time of the Dobosiewicz sort is as stated in the theorem. The lower 
bound on Shaker sort has a very similar proof, with the proviso that we require 2n bits to encode 
one pass of the algorithm rather than n bits. This results in the claimed lower bound of Q{n'^/4P) 
(which is nan- vacuous only for for p < ^ (log n — log log n) . □ 



6 Sorting with Queues and Stacks 

Knuth [11] and Tarjan [27] have studied the problem of sorting using a network of queues or 
stacks. The main variant of the problem is as follws: Given that the stacks or queues are arranged 
sequentially as shown in Figure 1, or in parallel as shown in Figure 2. Question: how many stacks 
or queues are needed to sort n elements with comparisons only? We assume that the input sequence 
is scanned from left to right, and the elements follow the arrows to go to the next stack or queue 
or output. In [12, 14] only the average-case analyses of the above two main variants was given, 
although the technique applies more in general to arbitrary acyclic networks of stacks and queues 
as studied in [27]. 

ouSm Input Permutation 

A sequence of stacks or queues 
Figure 1: Six stacks/queues arranged in sequential order 
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Sorted 
output 




Input Permutation 



stacks/queues 



Figure 2: Six stacks/queues arranged in parallel order 



6.1 Sorting with Sequential Stacks 

The sequential stack sorting problem is given in [11] exercise 5.2.4-20. We have k stacks numbered 
Sq, . . . , Sk-i- The input is a permutation tt of the elements 1, . . . , n. Initially wc push the elements 
of TT on 5o, at most one at a time, in the order in which they appear in vr. At every step we can 
pop a stack (the popped elements will move left in Figure 1) or push an incoming element on a 
stack. The question is how many stack are needed for sorting tt. It is known that k = log n stacks 
suffice, and ^logn stacks are necessary in the worst-case [11, 27]. Here we prove that the same 
lower bound also holds on the average, using a very simple incompressibility argument. 

Theorem 4 On average (uniform distribution), at least ^logn stacks are needed for sequential 
stack sort. 



Proof. Fix a random permutation tt such that 

C(7r|n, P) > logn! — logn = nlogn — 0{n), 

where P is an encoding program to be specified in the following. 

Assume that k stacks are sufficient to sort vr. We now encode such a sorting process. For every 
stack, exactly n elements pass through it. Hence we need perform precisely n pushes and n pops on 
every stack. Encode a push as and a pop as 1. It is easy to prove that different permutations must 
have different push/pop sequences on at least one stack. Thus with 2kn bits, we can completely 
specify the input permutation tt. Then, as before, 

2A;n > log n\ — log n = n log n — 0{n). 

Therefore, we have k > ^ logn — 0(1) for the random permutation tt. 

Since most (a (1 — l/n)th fraction) permutations are incompressible, we calculate the average- 
case lower bound as: 

1 n-1 ^ 1 1 
-logn h 1 • - ~ Trlogn. 

2 n n 2 

□ 
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6.2 Sorting with Parallel Stacks 

Clearly, the input sequence 2, 3, 4, . . . , n, 1 requires n — 1 parallel stacks to sort. Hence the worst- 
case complexity of sorting with parallel stacks, as shown in Figure 2, is n — 1. However, most 
sequences do not need this many stacks to sort in a parallel arrangement. The next two theorems 
show that on average, Q{^/n) stacks are both necessary and sufficient. Observe that the result 
is actually implied by the connection between sorting with parallel stacks and longest increasing 
subsequences in [27], and the bounds on the length of longest increasing subsequences of random 
permutations given in [9, 17, 8]. However, the proofs in [9, 17, 8] use deep results from probability 
theory (such as Kingman's ergodic theorem) and are quite sophisticated. Here we give simple proofs 
using incompressibility arguments. 

Theorem 5 On average (uniform distribution), the number of parallel stacks needed to sort n 
elements is 0{-\/n). 

Proof. Consider an incompressible permutation tt satisfying 

C(7r|n) > logn! — logn. (9) 

We use the following trivial algorithm (described in [27]), to sort tt with stacks in the parallel 
arrangement shown in Figure 2. Assume that the stacks are So,Si,. . ., and the input sequence is 
denoted as xi, . . . , x„. 

Algorithm Parallel- Stack- Sort 

1. For z = 1 to n do 

Scan the stacks from left to right, and push Xi on the the first stack Sj whose top element 
is larger than Xi. If such a stack doesn't exist, put Xi on the first empty stack. 

2. Pop the stacks in the ascending order of their top elements. 

We claim that algorithm Parallel-Stack-Sort uses 0(-y/n) stacks on the permutation vr. First, we 
observe that if the algorithm uses m stacks on tt then we can identify an increasing subsequence of 
TT of length m as in [27]. This can be done by a trivial backtracking starting from the top element 
of the last stack. Then we argue that tt cannot have an increasing subsequence of length longer 
than e^/n, where e is the natural constant, since it is compressible by at most logn bits. 

Suppose that a is a longest increasing subsequence of vr and m = \a\ is the length of a. Then 
we can encode tt by specifying: 

1. a description of this encoding scheme in 0(1) bits; 

2. the number m in logm bits; 

3. the combination a in log (^) bits; 

4. the locations of the elements of cr in tt in at most log (^) bits; and 

5. the remaining tt with the elements of a deleted in log(n — m)\ bits. 
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This takes a total of 

7x1 

log(n — m)! + 2 log — -- — '■ — + log m + 0(1) + 2 log log m 

m'.{n — m)l 

bits, where the last log log m term serves to self-delimitingly encode m. Using Stirling's approxi- 
mation, and the fact that ^/n < m = o(n), the above expression is upper bounded by: 

(n/e)" 

log n! + log - — --75 — 77 -— h Oflog n) 

ft Tl 

log n\ + m log — K + in — m) log \- m log e + 0(log n) 

n — m 

Tl 

log n! + m log — ^ + 2mloge + O(logn) 

This description length must exceed the complexity of the permutation which is lower-bounded 
in (9). Therefore, approximately m < e^/n, and hence m = 0{^/n). Hence, the average complexity 
of Parallel-Stack-Sort is 

r? — 1 1 

O(V^) • + n • - = O(v^). 

n n 

□ 



Theorem 6 On average (uniform distribution), the number of parallel stacks required to sort a 
permutation is n{^/n). 

Proof. Let ^ be a sorting algorithm using parallel stacks. Fix a random permutation vr with 
C{n\n, P) > logn! — logn, where P is the program to do the encoding discussed in the following. 
Suppose that A uses T parallel stacks to sort it. This sorting process involves a sequence of moves, 
and we can encode this sequence of moves by a sequence of instructions of the types: 

• push to stack i, 

• pop stack j, 

where the element to be pushed is the next unprocessed element from the input sequence, and the 
popped element is written as the next output element. Each of these term requires logT bits. In 
total, we use precisely 2n terms since every element has to be pushed once and has to be popped 
once. Such a sequence is unique for every permutation. 

Thus we have a description of an input sequence in 2n log T bits, which must exceed C{Tr\n, P) > 
nlogn — O(logn). It follows that T > ^/n = ^l{^/n). 

This yields the average-case complexity of A: 

r? — 1 1 

j^(v^) • - — + 1 • - = n{V7i). 

n n 

□ 



6.3 Sorting with Parallel Queues 

It is easy to see that sorting cannot be done with a sequence of queues. So we consider the complexity 
of sorting with parallel queues. It turns out that all the result in the previous subsection also hold 
for queues. 
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As noticed in [27], the worst-case complexity of sorting with paraUel queues is n, since the 
input sequence n,n — requires n queues to sort. We show in the next two theorems that 

on average, 0{-\/n) queues are both necessary and sufficient. Again, the result is implied by the 
connection between sorting with parallel queues and longest decreasing subsequences given in [27] 
and the bounds in [9, 17, 8] (with sophisticated proofs). Our proofs are trivial given the proofs in 
the previous subsection. 

Theorem 7 On average (uniform distribution), the number of parallel queues needed to sort n 
elements is upper bounded by 0{^/n). 

Proof. The proof is very similar to the proof of Theorem 5. We use a slightly modified greedy 
algorithm as described in [27]: 

Algorithm Parallel-Queue-Sort 

1. For i = 1 to n do 

Scan the queues from left to right, and append Xi on the the first queue whose rear element 
is smaller than Xj. If such a queue doesn't exist, put Xi on the first empty queue. 

2. Delete the front elements of the queues in the ascending order. 

Again, wc claim that algorithm Parallcl-Queue-Sort uses 0{^/rl) queues on every permutation 
TT, that cannot be compressed by more than log n bits. We first observe that if the algorithm uses 
m queues on tt then a decreasing subsequence of tt of length m can be identified, and we then argue 
that TT cannot have a decreasing subsequence of length longer than e^/n, in a way analogous to the 
argument in the proof of Theorem 5. 

□ 

Theorem 8 On average (uniform distribution), the number of parallel queues required to sort a 
permutation is ft{'s/n). 

Proof. The proof is the same as the one for Theorem 6 except that we should replace "push" 
with "enqueue" and "pop" with "dequeue". □ 
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